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DISPLAYING HETEROGENEOUS VIDEO 
Background 

This invention relates generally to video display 
devices and particularly to display devices that display 
heterogeneous video . 
5 Heterogeneous video is video from disparate sources 

which is intended to be displayed on a single video display 
device. The video may be graphics or streaming video 
normally associated with television programming. The 
graphics may come, for example, from an associated 

10 processor-based system for display on a display device 

which also receives other video sources. The heterogeneous 
video may also include video from a playback device such as 
a video cassette recorder or digital versatile disk player, 
games, and applications like e-mail, web browsers and word 

15 processors. 

Conventionally, heterogeneous video is displayed on a 
single display device by mixing the disparate content 
within a processor-based system and then coding the 
disparate content to a single common video output signal 

20 for interface to the display. Inevitably, such an output 
represents an awkward compromise between different ideal 
representations for each of the sources and limitations 
imposed by the actual display device. 
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For example, the simultaneous display of a first video 
at twenty- four bits per pixel and sixty frames per second 
with a second video at sixteen bits per pixel and sixty 
frames per second may require that both videos be converted 
5 into a common output format, for example, of twenty- four 
bits per pixel and sixty frames per second. While this 
output format may be advantageous for the first video, it 
amounts to a over representation of the second video which 
only requires sixteen bits per pixel. This u up- conversion" 

10 of the second video source unnecessarily increases the 

amount of bandwidth required to transport the first source 
to the display. 

The unnecessary additional bandwidth and sub-optimal 
representation of at least one of more than one video 

15 source may result from the practice of adapting 

heterogeneous video sources to a single format. Thus, 
there is a need for a way of handling heterogeneous video 
which does not unnecessarily waste bandwidth or diminish 
the representation of some (if not all) of the video from 

2 0 various video sources. 

Brief Description of the Drawings 
Figures 1A and IB are schematic depictions of one 
embodiment of the present invention including a computing 
device which communicates with a display device; 
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Figure 2 is a screen display for the embodiment shown 
in Figures 1A and IB in accordance with one embodiment of 
the present invention; 

Figure 3 is a schematic depiction corresponding to 
5 Figure IB for still another embodiment of the present 
invention; 

Figure 4 is a schematic cross -sectional view of one 
embodiment of the display in accordance with the present 
invention; 

10 Figure 5 is a more detailed, enlarged cross-sectional 

view of the display of the type shown in Figure 1; 

Figure 6 is a schematic diagram of one cell of the 
display in accordance with another embodiment of the 
present invention; 
15 Figure 7 is a schematic diagram of a display 

implementing one embodiment of the present invention; and 

Figure 8 is a flow chart for software in accordance 
with one embodiment of the present invention. 

Detailed Description 
2 0 Referring to Figure 1, a computing device 12 may 

communicate over an interface 34 with a display device 14 . 
In one embodiment of the present invention, the interface 
34 is a wireless interface such as a radio wave or infrared 
interface. However, other interfaces including wired 
25 connections may be utilized as well. 
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In one embodiment of the present invention, the 
computing device 12 may be a set -top box that communicates 
with a display device 14 that is a television receiver. 
However, the computing device 12 may be any processor-based 
5 device including a desktop computer, a laptop computer, or 
a processor-based appliance. 

In accordance with one embodiment of the present 
invention, when a heterogeneous video is intended to be 
displayed on the display device 14, each of the disparate 

10 video streams (such as the sources 1-4 in Figure 1) that 
make up the heterogeneous video may be conveyed from the 
computing device 12 to the display device 14 independently. 
These video streams come from a variety of sources, 
including a television signal or graphics generated by the 

15 processing unit of the computing device 12. 

The computing device 12 may include a central 
processing unit 18, graphics or other processing units 26, 
a television receiver 22, and other input devices 24 
coupled by a bus 16. A system memory 20 may also be 

2 0 coupled to the bus 16. A video controller 28, coupled to 
the bus 16, provides a series of independent video sources, 
indicated as sources 1-4 in Figure 1, to an arbitration and 
packetization unit 30. 

As an example, a first video stream (source 1) may 

25 come from a terrestrial or satellite television system 
while a second video stream (source 2) may represent 
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graphical information describing the first video source and 
generated locally by the computing device 12 . These 
graphics may be a television programming guide also known 
as an electronic content guide or an electronic programming 
5 guide (EPG) . 

The arbitration and packetization unit 3 0 may 
simultaneously drive independent video sources at their 
natural rates onto different portions of the display screen 
of a display device 14. Thus, in the example described 

10 above, the video information may be displayed on one 

portion of the display 14 at a native frame rate of sixty 
Hertz while the graphical information may be updated more 
infrequently, for example at 25 Hertz. 

Each of the video sources may be converted into 

15 packets by the unit 30 for more efficient transport by a 
transport modulation unit 32. If more sources are 
available than the bandwidth permits, the unit 3 0 
arbitrates which sources are transmitted during a given 
interval . 

20 In the display device 14, the received packets from 

the interface 34 are demodulated by a transport 
demodulation unit 36. The demodulated packets are then 
depacketized in the depacketization unit 38. The unit 38 
recreates the separate video sources which are then 

25 converted by the conversion unit 4 0 into an appropriate 
format for display on the display element 42. 
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In one embodiment of the present invention, the 
display screen 42 may use liquid crystal over semiconductor 
(LCOS) imaging devices with embedded storage and other 
processing elements. The conversion unit 40 receives the 
5 independent video signals, converts the signals to a format 
compatible with its LCOS display element or screen 42 and 
then drives the signals onto the LCOS display screen 42. 
By incorporating an integrated storage into the LCOS 
display screen 42, as described hereinafter, the video 

10 signals may be driven onto the display element 42 in a 

random access pattern similar to that employed when driving 
new information into traditional memory technologies such 
as dynamic random access memory (DRAM) and static random 
access memory (SRAM) . 

15 Thus, referring to Figure 2, an electronic programming 

guide (EPG) may be displayed in the region 46 of the LCOS 
display screen 42 and an ongoing television program may be 
displayed in the region 44 in one embodiment of the 
invention. The EPG may include a plurality of channels 

20 listed down the left margin and a series of times, listed 
across the top of the region 4 6 in a grid pattern 
indicating what programs (shows # 1-7) are available for 
viewing on any given channel at any of the indicated times. 
Thus, the user may select from among the electronic 

25 programming guide entries. For example, by mouse clicking 
using an input device, a user may select a particular 
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program for full screen viewing. While making the 
decision, one video source may be displayed in the region 
44. Of course, a wide variety of display formats may be 
used. 

5 In this way, the video stream from one video source 

may be displayed at its native rate in the region 44 while 
the electronic programming guide (which may be in the form 
of graphical information) displayed in the region 46 may be 
updated less frequently. Thus, each of the video sources 
10 may be assigned to a distinct, predetermined region of the 
overall display screen 42 to facilitate the accommodation 
of the native characteristics of each of two or more video 
sources . 

While Figure 2 illustrates a situation where only two 
15 video sources are displayed, more than two video sources 
may be provided by dedicating specific screen portions to 
each video source. In one embodiment of the present 
invention, the display screen 42 may be divided into 
quadrants each assigned to one of four potential video 
2 0 sources. In some cases, where only one source is 

available, that source may be displayed full screen. In 
other cases, the user may select among the various 
available sources for either full screen or split screen 
display. 

25 In some embodiments of the present invention, the 

video streams that drive less information to the display 
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than others need not be "up-converted" to higher 
information formats prior to transmission to the display. 
This may save transmission bandwidth between the computing 
device 12 and the display device 14 that might otherwise be 
5 used for the transport of other valuable video information. 
In addition, the ability to present video sources at their 
native rates may yield a perceptually superior video 
presentation. The rate conversion process may degrade the 
source from its native format. Because the burden of 

10 converting all video sources into a single common format 
may be removed from the computer device 12 and the 
bandwidth of the information being driven to the display 
may potentially be reduced, the requirements on the memory 
and processing elements in the computer device 12 may be 

15 eased, yielding a system with additional performance 

margin. The added performance margin may then be applied 
to other processing operations or alternatively, the system 
may be cost reduced to a level sufficient to simply drive 
the display. 

2 0 The display element 42 may be implemented as an 

electro-optical device 110, such as a spatial light 
modulator (SLM) as shown in Figure 4. The device 110 may 
include a plurality of reflective mirrors 112 defined on a 
semiconductor substrate 114 in accordance with one 

2 5 embodiment of the present invention. Advantageously, the 
device 110 is implemented using liquid crystal over 
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semiconductor (LCOS) technology. LCOS technology may form 
large screen projection displays or smaller displays (using 
direct view rather than projection technology) . With LCOS 
technology, the liquid crystal display is formed in 
5 association with the same substrate that forms 

complementary metal oxide semiconductor (CMOS) circuit 
elements. The display may be a reflective liquid crystal 
display. 

The device 110 may include a silicon substrate 114 

10 with a metal layer defining the mirrors 112 . The mirrors 
112 may be the mirrors of an electro-optic display such as 
a liquid crystal display. For example, the mirrors 112 may 
be part of spatial light modulator (SLM) for one of the 
color planes of a tricolor display. Potentials applied to 

15 the mirrors 112 alter the liquid crystal to modulate the 

incoming light to create images which then can be directly 
viewed or projected onto a projection screen. 

Referring to Fig. 5, each cell or pixel of the display 
may include a reflective mirror 124 forming one of the 

20 mirrors of one of the pixels 112 shown in Fig. 4. In one 
embodiment of the invention, each cell may be rectangular 
or square and a slight spacing may occur between each 
adjacent mirror 124. Thus, a rectangular array of mirrors 
124 may form an array of pixel elements in conjunction with 

25 liquid crystal material 120 positioned over the mirrors 
124 . 
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The LCOS structure includes a substrate 114 having 
doped regions 132 formed therein. The doped regions 132 
may define transistors for logic elements and/or memory 
cells which operate in conjunction with the display pixels 
5 as will be described hereinafter. Four or more metal 

layers may be provided, including a metal one layer 13 0 
which is spaced by an inter-layer dielectric (ILD) 131 from 
a metal two layer 128 and a metal three layer 126. A metal 
four layer may form the pixel mirrors 124. Thus, for 

10 example, the metal two layer 12 8 may provide light blocking 
and the metal one layer may provide the desired 
interconnections for forming the semiconductor logic and 
memory devices. The pixel mirrors 124 may be coupled, by 
way of vias 133, with the other metal layers. 

15 A dielectric layer 122 may be formed over the mirror 

124. A liquid crystal or electro-optic material 120 is 
sandwiched between a pair of buffered polyimide layers 119a 
and 119b. One electrode of the liquid crystal device is 
formed by the metal layer 124. The other electrode is 

20 formed by an indium tin oxide (ITO) layer 118. 

A top plate 116 may be formed of transparent material. 
The ITO layer 118 may be coated on the top plate 126. The 
polyimide layers 119a and 119b provide electrical isolation 
between the capacitor plates which sandwich the electro- 

25 optic material 120. However, other insulating materials 
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may be coated on the ITO layer 118 in place of or in 
addition to the polyimide layers. 

Using the LCOS structure, for example as depicted in 
Fig. 6, a memory element or array may be incorporated into 
5 the same silicon substrate which includes the pixel array. 
A memory 160 may be integrated with each pixel cell 112. 
In some embodiments, pixel information may be passed 
through a digital to analog converter (DAC) 162 to produce 
gray scale information. The particular manner in which 

10 pixels are arranged in the storage array and converted to 
analog signals may vary by implementation. 

Each pixel cell 112 metal electrode or top metal 124 
may be coupled to a separate DAC 162. In one embodiment of 
the present invention, the DAC may be an eight bit DAC 

15 coupled to eight one bit storage elements 160. Each 

storage memory 160 may, for example, be a static random 
access memory (SRAM) cell. Each one bit storage element 
160 may be coupled by a transfer transistor 158 to a 
different row 156 and a column 154. Thus, the information 

2 0 which is used to refresh the metal 124 may be stored in the 
memory 160. When it is desired to change the pixel 
information to change the displayed image, then the 
information in the memory 160 is refreshed. 

Since the display refresh controller only needs to 

25 refresh new information to the display 14, the overall 

drain on the computing device 12 including the buses and 
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memory may be reduced, potentially yielding better 
performance out of the other components in the computer 
system which rely on these limited resources. In addition, 
the amount of redundant information flowing to the display 
5 14 may be reduced, allowing more new information to be sent 
to the display. This potentially enables the display of 
higher resolution or higher rate images . 

In one embodiment of the present invention, a 
projection display 164, shown in Figure 7, includes the 

10 spatial light modulator display panels 166, 174 and 176, 
using liquid crystal over silicon technology with 
integrated memory. The reflective liquid crystal display 
projection system 614 typically includes a modulator or 
display panel (LCD display panels 174, 166 and 176) for 

15 each primary color that is projected onto a screen 192. In 
this manner, for a red-green-blue (RGB) color space, the 
projection system 164 may include an LCD display panel 174 
that is associated with a red color band, an LCD display 
panel 166 that is associated with the green color band and 

20 LCD display panel 176 that is associated with the blue 

color band. Each of the LCD display panels 166, 174 and 
176 modulates light from the light source 194 and the 
optics 196 that form red, green and blue images, 
respectively, and add together to form a composite color 

25 image on the screen 192. To accomplish this, each LCD 
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display panel receives electrical signals indicating the 
corresponding modulated beam image to be formed. 

More particularly, the projection display 64 may 
include a beam splitter 186 that directs a substantially 
5 collimated white beam 198 of light, provided by the light 

source 194, to optics that separate the white beam 198 into 
red 182, blue 178 and green 2 02 beams. In this manner, the 
white light beam 198 may be directed to a red dichroic 
mirror 172 that reflects the red beam towards the LCD 

10 display panel 174 that, in turn, modulates the red beam 

182. The blue beam passes through the red dichroic mirror 
to a blue dichroic mirror 170 that reflects the blue beam 
towards the LCD panel 176 for modulation. The green beam 
2 02 passes through the red and blue dichroic mirrors for 

15 modulation by the LCD display panel 166, 

For reflective LCD display panels, each LCD display 
panel 166, 174 and 176 modulates the incident beam and 
reflects the modulated beams 168, 200 and 190 respectively, 
so that the modulated beams return on the paths described 

2 0 above to the beam splitter 186. The beam splitter 186, in 
turn, directs the modulated beams through projection optics 
such as a lens 188, to form modulated beam images that 
ideally overlap and combine to form the composite image on 
the screen 192. Each of the panels 166, 174, and 176 may 

25 be implemented using liquid crystal over semiconductor 
technology as illustrated for example in Fig. 5. 
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Referring to Figure 3, another embodiment of the 
present invention may include a computing device 12 which 
operates as described in connection with Figure 1A and a 
display device 14a which has been modified from the 
5 embodiment shown in Figure IB. 

The display device 14a may use a legacy display 
technology such as a cathode ray tube or a thin film 
transistor-liquid crystal display 52 with integrated memory 
and processing elements. Like the embodiment of Figure 1A, 

10 independent video streams may be driven to the display 

device 14a from the computing device 12 . In the display 
device 14a, these video sources are converted to a common 
frame buffer format, and driven onto the integrated memory 
48 through the use of an arbiter and addressing circuitry 

15 40a. Simultaneously, video is scanned out of this memory 

48 in a synchronous, regular fashion by a controller 50 and 
driven onto the imaging elements 52 . 

In accordance with one embodiment of the present 
invention, software 210, shown in Figure 8, may be utilized 

20 by the computing device 12 in the embodiment of Figures 1A 
to control the generation of independent, packetized video 
sources. At least two video streams are received in the 
computing device 12 as indicated in block 212. Each video 
stream is packetized as indicated in block 214. The 

25 independent, packetized video streams are then sent to the 
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display as indicated in block 216 for subsequent de- 
packetization and display. 

Some embodiments of the present invention may exhibit 
superior performance to systems where heterogeneous video 
5 signals are mixed in the computing device prior to 

modulation for the display. Since only new information is 
transferred to the display, bandwidth may be saved, power 
consumption may be reduced and the generation of heat in 
the communication between the computing device and the 

10 display may be reduced in some embodiments. In addition, 
in some embodiments of the present invention, perceptually 
superior display of disparate video sources is possible 
relative to systems where the content is mixed in the 
computing device. Instead of aggregating and synchronizing 

15 all the video sources in the computing device and thereby 

forcing all streams into a single, least-common-denominator 
format and timing, video sources may be independently 
streamed to the display and presented in their native 
format. In some cases, the performance of the computing 

20 device may be enhanced. 

While the present invention has been described with 
respect to a limited number of embodiments, those skilled 
in the art will appreciate numerous modifications and 
variations therefrom. It is intended that the appended 

25 claims cover all such modifications and variations as fall 
within the true spirit and scope of this present invention. 
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1 1. A method comprising: 

2 streaming at least two independent video sources 

3 for display on a video display screen; and 

4 causing said sources to be displayed at separate 

5 regions of said display screen. 

1 2 . The method of claim 1 including forming said 

2 sources into packets in a first device and transporting 

3 said packets to a second device. 

1 3. The method of claim 2 including depacketizing 

2 said packets in said second device. 

1 4. The method of claim 1 including transmitting said 

2 video sources from a processor-based system to a display 

3 device including said display screen. 

1 5 . The method of claim 4 including transmitting said 

2 video sources over a wireless connection between said 

3 processor-based system and said display device. 

1 6. The method of claim 1, wherein said display 

2 screen includes a pixel array and a memory array, 

3 refreshing said memory array and said pixel array in the 

4 same refresh cycle. 
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1 7 . The method of claim 6 including displaying said 

2 sources on a display that uses liquid crystal over 

3 semiconductor technology. 

1 8 . The method of claim 1 including streaming video 

2 sources for display on said display screen at different 

3 frame rates . 

1 9 . The method of claim 1 wherein one of said video 

2 sources includes television programming and the other of 

3 said video sources includes graphical information. 

1 10. The method of claim 1 including streaming a first 

2 video source that includes television programming 

3 information and a second video source that includes an 

4 electronic programming guide information. 

1 11. A system comprising: 

2 a processor; 

3 storage coupled to said processor; 

4 a video controller coupled to said processor; 

5 a packetization device coupled to said video 

6 controller which independently packetizes at least two 

7 video streams . 
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1 12. The system of claim 11 including a modulation 

2 device to modulate and transport said independently 

3 packetized streams. 

1 13. The system of claim 11 wherein each of said video 

2 streams has a different frame rate and is packetized to be 

3 de-packetized at the original frame rate in a display 

4 device . 

1 14 . An article comprising a medium storing 

2 instructions that cause a processor-based system to: 

3 receive two independent video sources; and 

4 packetize each of said video sources so that they 

5 may be displayed in separate regions of a display screen. 

1 15. The article of claim 14 further storing 

2 instructions that cause a processor-based system to 

3 transmit said video sources from said processor-based 

4 system to a display device including a display screen. 

1 16. The article of claim 15 further storing 

2 instructions that cause the processor-based system to 

3 transmit said video sources over a wireless connection 

4 between said processor-based system and said display 

5 device. 
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1 17. The article of claim 16 further storing 

2 instructions that cause the processor-based system to 

3 transmit said video sources for display on said display 

4 screen at different frame rates. 

1 18. A system comprising: 

2 a semiconductor substrate; 

3 a liquid crystal over semiconductor pixel array 

4 formed in said substrate; 

5 a memory coupled to said array, said memory also 

6 formed in said substrate; and 

7 a device for receiving a signal made up of a 

8 plurality of independent video sources and driving each of 

9 said video sources for display on a different portion of 
10 said pixel array. 

1 19. The system of claim 18 wherein said system 

2 includes a device that de-packetizes said signal to form 

3 independent video sources for display on said pixel array. 

1 20. The system of claim 19 wherein said pixel array 

2 includes a plurality of pixels including a memory cell. 

1 21. The system of claim 20 wherein said memory cells 

2 are static random access memory cells. 
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1 22. The system of claim 19 wherein said pixel array 

2 is coupled to said memory by a digital to analog converter. 

1 23. The system of claim 19 wherein said memory 

2 includes a cell associated with each of a plurality of 

3 pixels of said pixel array. 

1 24. A system comprising: 

2 an imaging device having a plurality of imaging 

3 elements; 

4 a memory that receives and stores at least two 

5 independent video sources; and 

6 a controller that drives said video sources onto 

7 separate portions of said imaging device. 

1 25. The system of claim 24 wherein said imaging 

2 device is a thin film transistor imaging device. 

1 26. The system of claim 24 wherein said imaging 

2 device is a cathode ray tube. 

1 27. The system of claim 24 wherein said imaging 

2 device uses liquid crystal over semiconductor technology. 

1 28. The system of claim 24 including a device to 

2 receive packetized video information, de-packetize said 
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3 information and provide said de-packetized information to 

4 said memory as independent video sources. 
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DISPLAYING HETEROGENEOUS VIDEO 



Abstract of the Disclosure 
Heterogeneous video may be independently encoded in a 
processor-based system and transmitted for display on a 
display device. In the display device, the independent 
5 video streams may be de-packetized for display on the same 
display at different frame rates. Thus, each of the video 
sources may be displayed in a separate portion of the 
display at its native frame rate. For example, an 
electronic programming guide may be transmitted for display 
10 at one frame rate and displayed in a distinct region on the 
display screen while video corresponding to an ongoing 
television program may be displayed in another portion of 
the display screen at its native frame rate. 
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States provisional application^) listed below: 



(Application Number) (Filing Date) 



(Application Number) (Filing Date) 

I hereby claim the benefit under Title 35, United States Code, Section 120 of any United 
States application(s) listed below and, insofar as the subject matter of each of the claims 
of this application is not disclosed in the prior United States application in the manner 
provided by the first paragraph of Title 35, United States Code, Section 112, I 
acknowledge the duty to disclose all information known to me to be material to 
patentability as defined in Title 37, Code of Federal regulations, Section 1.56 which 
became available between the filing date of the prior application and the national or PCT 
International filing date of this application: 



(Application Number) Filing Date (Status-patented, pending, abandoned) 



(Application Number) Filing Date (Status-patented, pending, abandoned) 



I hereby appoint Timothy N. Trap, Reg. No. 28,994; Fred G. Pruner, Jr., Reg. No. 40,779 
and Dan C. Hu, Reg. No. 40,025 my patent attorneys, of TROP, PRUNER & HU, P.C., 
with offices located at 8554 Katy Freeway, Ste. 100, Houston, TX 77024, telephone (713) 
468-8880, and Joseph R. Bond, Reg. No. 36,458; Richard C. Calderwood, Reg. No. 
35,468; Sean Fitzgerald, Reg. No. 32,027; David J. Kaplan, Reg. No. 41,105; Leo V. 
Novakoski, Reg. No. 37,198; Naomi Obinata, Reg. No. 39,320; Thomas C. Reynolds, 
Reg. No. 32,488; Steven P. Skabrat, Reg. No. 36,279; Howard A. Skaist, Reg. No. 
36,008; Steven C. Stewart, Reg. No. 33,555; Raymond J. Werner, Reg. No. 34,752; and 
Charles K. Young, Reg. No. 39,425; my patent attorneys, of INTEL CORPORATION; with 
full power of substitution and revocation, to prosecute this application and to transact all 
business in the Patent and Trademark Office connected herewith. 

Send correspondence to Timothy N. Trop . TROP, PRUNER & HU, P.C., 8554 Katy 
Freeway, Ste. 100, Houston, TX 77024 and direct telephone calls to Timothy N T Trop, 
(713)468-8880. 

I hereby declare that all statements made herein of my own knowledge are true and that 
all statements made on information and belief are believed to be true; and further that 
these statements were made with the knowledge that willful false statements and the like 
so made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of 
the United States Code and that such willful false statements may jeopardize the validity of 
the application or any patent issued thereon. 
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Full Name of Sole/First Inventor: 
SCOTT A. ROSENBERG 


Inventor's^^^^^^ j^^^ 


Date: / / 


Residence: / 
CUPERTINO, CALIFORNIA / 


Citizenship: 
U.S. 


Post Office Address: / 

19312 GREENWOOD DRIVE, CUPERTINO, CALIFORNIA 95014 


Full Name of Second/Joint Inventor: 
OLEG B. RASHKOVSKIY 


Inventor's Signature: /V 


Date: 


Residence: 

PALO ALTO, CALIFORNIA 


Citizenship: 
U.S. 


Post Office Address: 

875 UNIVERSITY AVE. #9, PALO ALTO, CALIFORNIA 94301 



lNTL-0320 -US (P8003) 
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